Image pickup device, flash image generating method and computer-readable memory medium

ABSTRACT

When a displacement between a reference frame of a plurality of images acquired by continuous image pickup and a target frame is less than a first threshold indicating that such a frame is not likely to be affected by occlusion, smoothing is performed on an object area through a morphological operation with a normal process amount. Conversely, when a displacement between the reference frame of the plurality of images acquired by continuous image pickup and a target frame is larger than or equal to the first threshold, smoothing is performed with the process amount of morphological operation being increased with respect to the normal process amount.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a Divisional of U.S. application Ser. No. 12/638,021, which isbased upon and claims the benefit of priority from prior Japanese PatentApplications No. 2008-322960, filed Dec. 18, 2008, and No. 2009-143736,filed Jun. 16, 2009, the entire contents of all of which areincorporated herein by reference.

FIELD

The present invention relates to an image pickup device suitable forgenerating a flash image, a flash image generating method, and acomputer-readable memory medium.

BACKGROUND

There are known flash images like one shown in FIG. 22 that images of amoving object are picked up by a fixed camera and how the object movesis expressed in one image (picture). According to film cameras, strobeflashing is carried out plural times relative to a moving object duringa long-time exposure to generate such a flash image. In recent days thatdigital cameras become widespread, flash images can be generated throughan image processing by a computer in cameras.

In generating a flash image, images of a moving object are picked up bycontinuous image-pickup with the angle of view of a digital camera and adirection thereof being set at constant. When images are picked up withthe digital camera being held in hands, because the digital cameraslightly moves, a change in the angle of view among continuouslypicked-up frame images may be caused. In a case in which the depth of abackground of an object (a moving object) is large, when thedisplacement between the frame images is large, because of occlusion, adifference in a background area is created which must be originally stayat the same place among the frame images. If such a difference is large,the background part may be falsely recognized as the moving object. As aresult, the background part is duplicated (duplicatingly synthesized)over a generated flash image.

SUMMARY

The present invention has been made in view of the foregoingcircumstance, and it is an object of the present invention to provide animage pickup device which can generate a better flash image, a flashimage generating method, and a computer-readable recording medium.

To achieve the object, an image pickup device according to the firstaspect of the present invention comprises: an image pickup unit whichgenerates a plurality of continuously picked-up images by image pickup;an object extracting unit which extracts a plurality of images eachrepresenting a moving object part from individual picked-up imagesgenerated by the image pickup unit; a background image generating unitwhich generates a background image from the plurality of picked-upimages generated by the image pickup unit; a flash image generating unitwhich synthesizes the background image generated by the background imagegenerating unit with the plurality of images each representing theobject part extracted by the object extracting unit to generate a flashimage; a displacement detecting unit which detects a displacementbetween predetermined images among the plurality of picked-up images;and an image smoothing unit which performs smoothing on the image by aprocess amount in accordance with the displacement detected by thedisplacement detecting unit.

To achieve the object, a flash image generating method according to thesecond aspect of the present invention which generates a flash imagefrom a plurality of continuous images in which images of a moving objectare picked up using a device which executes an image processing, themethod comprises; an object extracting step of causing the device toextract a plurality of images each representing the moving object partfrom the plurality of continuous images; a background image generatingstep of causing the device to generate a background image from theplurality of continuous images; a displacement detecting step of causingthe device to detect a displacement between predetermined images amongthe plurality of continuous images; an image smoothing step of causingthe device to perform smoothing on an image extracted in the objectextracting step by a process amount in accordance with the displacementdetected in the displacement detecting step; and a flash imagegenerating step of causing the device to synthesize the background imagegenerated in the background image generating step with the plurality ofimages each representing the object part extracted in the objectextracting step to generate a flash image.

To achieve the object, a computer-readable memory medium according tothe third aspect of the present invention stores a program which allowsa computer to realize: a function of acquiring a plurality of continuousimages in which images of a moving object are picked up; a function ofextracting a plurality of images each representing a moving object partfrom the plurality of continuous images; a function of generating abackground image from the plurality of continuous images; a function ofdetecting a displacement between predetermined images among theplurality of continuous images; a function of performing smoothing onthe extracted image by a process amount in accordance with the detecteddisplacement; and a function of synthesizing the generated backgroundimage with the plurality of images each representing the object part togenerate a flash image.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of this application can be obtained whenthe following detailed description is considered in conjunction with thefollowing drawings, in which:

FIG. 1 is a block diagram showing a configuration of a digital cameraaccording to an embodiment of the present invention;

FIG. 2 is a function block diagram showing functions realized by acontrol unit in FIG. 1;

FIG. 3 is a flowchart for explaining a “flash image generating process”according to the embodiment of the present invention;

FIG. 4A is a drawing for explaining a display example of a settingscreen displayed in the “flash image generating process” in FIG. 3, andshowing a display example of the setting screen for specifying adirection of the camera;

FIG. 4B is a drawing for explaining a display example of a settingscreen displayed in the “flash image generating process” in FIG. 3, andshowing a display example of the setting screen for specifying a movingdirection of an object;

FIG. 5A is a drawing for explaining a picked-up image according to theembodiment of the present invention, and exemplifying a scene ofcontinuous image-pickup necessary for generating a flash image;

FIG. 5B is a drawing for explaining a picked-up image according to theembodiment of the present invention, and showing an example of apicked-up image (continuously-picked-up image) acquired by continuousimage-pickup in FIG. 5A;

FIG. 6 is a flowchart for explaining an “alignment process” executed inthe “flash image generating process” in FIG. 3;

FIG. 7 is a drawing showing an example of information stored in a “frameinformation table” created in the “alignment process” in FIG. 6;

FIG. 8 is a flowchart for explaining a “one-dimensional data generatingprocess” executed in the “flash image generating process” in FIG. 3;

FIG. 9A is a drawing for explaining the “one-dimensional data generatingprocess” in FIG. 8, and showing a relationship between a coordinate in apicked-up image and a moving direction of an object;

FIG. 9B is a drawing for explaining the “one-dimensional data generatingprocess” in FIG. 8, wherein (a) and (b) show an example of a projectionof a picked-up image and that of one-dimensional data, and (c) showsoverlapped one-dimensional image data shown in (a) and (b);

FIG. 10 is a flowchart for explaining an “object area detecting process”executed in the “flash image generating process” in FIG. 3;

FIG. 11 is a flowchart for explaining the “object area detectingprocess” executed in the “flash image generating process” in FIG. 3;

FIG. 12 is a flowchart for explaining the “object area detectingprocess” executed in the “flash image generating process” in FIG. 3;

FIG. 13A is a drawing for explaining the “object area detecting process”in FIGS. 10 to 12, and showing an example of an object area detectedwhen the moving direction of an object is X-direction;

FIG. 13B is a drawing for explaining the “object area detecting process”in FIGS. 10 to 12, and showing an example of an object area detectedwhen the moving direction of an object is Y-direction;

FIG. 14 is a flowchart for explaining an “effective frame selectingprocess” executed in the “flash image generating process” in FIG. 3;

FIG. 15 is a flowchart for explaining the “effective frame selectingprocess” executed in the “flash image generating process” in FIG. 3;

FIG. 16 is a drawing showing an example of a selected effective frame;

FIG. 17 is a flowchart for explaining a “background image generatingprocess” executed in the “flash image generating process” in FIG. 3;

FIG. 18 is a flowchart for explaining an “object image extractingprocess” executed in the “flash image generating process” in FIG. 3;

FIG. 19 a flowchart for explaining a “first image smoothing process”executed in the “flash image generating process” in FIG. 3;

FIG. 20 is a drawing for explaining how occlusion effects, and wherein(a) shows an example scene that is likely to be affected by occlusion,(b) shows an example frame image having a displacement in a backgroundpart, (c) shows an example of an original background image, (d) shows anexample background image deformed for alignment, (e) shows an example ofa difference caused between an original background image and a deformedbackground image, (f) is an example flash image affected by occlusion,and (g) shows an example of an unnecessary part falsely recognized as anobject image;

FIG. 21 is a flowchart for explaining an “image synthesizing process”executed in the “flash image generating process” in FIG. 3;

FIG. 22 is a drawing showing an example of a generated flash image;

FIG. 23 is a flowchart for explaining a “second image smoothing process”according to a second embodiment of the present invention; and

FIG. 24 is a flowchart for explaining a “third image smoothing process”according to a third embodiment of the present invention.

DETAILED DESCRIPTION

(First Embodiment) An explanation will be given of an embodiment of thepresent invention with reference to accompanying drawings. In a firstembodiment, an explanation will be given of an example case in which thepresent invention is embodied using a digital still camera (hereinafter,digital camera). A digital camera 1 of the embodiment has functions thatgeneral digital still cameras have, and such functions include at leasta so-called continuous image-pickup function. The continuousimage-pickup function in the embodiment means a function of acquiringplural continuous picked-up images through one shutter button operation.

The digital camera 1 of the embodiment also has a flash image generatingfunction of generating a flash image (multi-stroboscopic image) frompicked-up images acquired by the continuous image-pickup function as thepresent invention is applied. Note that a “flash image” means an imagethat a movement of a moving object is described in one image. In orderto generate a flash image using the digital camera 1 of the embodiment,images of a moving object are picked up by the continuous image-pickupfunction, and in this case, the direction of the digital camera 1 andthe angle of view thereof are set constant to pick up an image.

FIG. 1 is a block diagram showing a configuration of the digital camera1 according to the embodiment. As shown in FIG. 1, the digital camera 1of the embodiment comprises an image pickup unit 100, a data processingunit 200, an interface (I/F) unit 300, a sensor unit 400, and the like.

The image pickup unit 100 is a part which executes an image pickupoperation of the digital camera 1. As shown in FIG. 1, the image pickupunit 100 comprises an optical device 110, an image sensor unit 120, andthe like.

The optical device 110 includes, for example, a lens, a diaphragmmechanism, and a shutter mechanism. The optical device 110 executes anoptical operation with respect to image pickup. That is, as the opticaldevice 110 operates, incident light is collected, and optical factorsrelating to an angle of view, an exposure, or the like, such as a focalpoint distance, an aperture, and a shutter are adjusted. The shuttermechanism in the optical device 110 is a so-called mechanical shutter.When a shutter operation is carried out through merely an operation ofthe image sensor, the optical device 110 may not need the shuttermechanism. The optical device 110 operates under a control of thecontrol unit 210 to be described later.

The image sensor unit 120 comprises an image sensor, such as a CCD(Charge Coupled Device) or a CMOS (Complementally Metal OxideSemiconductor) which generates an electrical signal in accordance withincident light collected by the optical device 110. As the image sensorunit 120 performs photoelectric conversion, an electric signal inaccordance with reception of light is generated and is output to thedata processing unit 200.

The data processing unit 200 processes an electric signal generatedthrough an image pickup operation of the image pickup unit 100,generates digital data representing a picked-up image, and executes animage processing on the picked-up image. As shown in FIG. 1, the dataprocessing unit 200 comprises the control unit 210, an image processingunit 220, an image memory 230, an image output unit 240, a memory unit250, an external memory unit 260, and the like.

The control unit 210 comprises, for example, a processor like a CPU(Central Processing Unit), and a main memory device like a RAM (RandomAccess Memory). As the control unit 210 runs a program stored in thememory unit 250 to be described later, the control unit 210 controlseach unit of the digital camera 1. In the embodiment, as a predeterminedprogram is run, the control unit 210 realizes various functions relatingto individual processes to be described later.

The image processing unit 220 comprises, for example, an ADC(Analog-Digital Converter), a buffer memory, and a processor for animage processing (i.e., a so-called image processing engine). The imageprocessing unit 220 generates digital data representing a picked-upimage based on an electric signal generated by the image pickup unit100.

That is, the ADC converts an analog electric signal output by the imagesensor unit 120 into a digital signal, such digital signal issuccessively stored in the buffer memory, and the image processingengine performs a so-called development process on the buffered digitaldata, thus adjusting an image quality and compressing data.

The image memory 230 comprises a memory device, such as a RAM or a flashmemory. The image memory 230 temporarily stores picked-up image datagenerated by the image processing unit 220 and image data processed bythe control unit 210.

The image output unit 240 comprises a circuit generating an RGB signal.The image output unit 240 converts image data stored in the image memory230 into an RGB signal, and outputs such an RGB signal to a displayscreen (e.g., a display unit 310 to be described later).

The memory unit 250 comprises a memory device, such as a ROM (Read OnlyMemory) or a flash memory. The memory unit 250 stores program and datanecessary for operating the digital camera 1. In the embodiment, thememory unit 250 stores an operation program executed by the control unit210, parameters and arithmetic expressions necessary for executingprocesses.

The external memory unit 260 comprises a memory device removable fromthe digital camera 1 like a memory card. The external memory unit 260stores image data picked up by the digital camera 1.

The interface unit 300 functions as an interface between the digitalcamera 1 and a user or an external device. As shown in FIG. 1, theinterface unit 300 comprises the display unit 310, an external interface(I/F) unit 320, an operation unit 330, and the like.

The display unit 310 comprises, for example, a liquid crystal displaydevice. The display unit 310 displays and outputs various screensnecessary for operating the digital camera 1, a live-view image at thetime of image pickup, a picked-up image, and the like. In theembodiment, a picked-up image or the like is displayed and output basedon an image signal (an RGB signal) from the image output unit 240.

The external interface unit 320 comprises, for example, a USB (UniversalSerial Bus) connector, and a video output terminal. The externalinterface unit 320 outputs image data to an external computer device,and outputs and displays a picked-up image on an external monitordevice.

The operation unit 330 comprises various buttons provided at an externalface of the digital camera 1. The operation unit 330 generates an inputsignal corresponding to an operation by the user of the digital camera1, and inputs such a signal into the control unit 210. The buttonsconfiguring the operation unit 330 include, for example, a shutterbutton for instructing a shutter operation, a mode button for specifyingan operation mode of the digital camera 1, a cross key and a functionbutton for various setting, and the like.

The sensor unit 400 is provided in the digital camera 1 if the digitalcamera 1 has a camera-shake compensation function. The sensor unit 400detects a movement of the digital camera 1. The sensor unit 400comprises, for example, a gyro sensor. The sensor unit 400 detects amovement of the digital camera 1 caused by depression of the shutterbutton for example, and inputs a detection value to the control unit210.

In the embodiment, as the control unit 210 runs the operation programstored in the memory unit 250, individual processes to be describedlater are executed. Functions realized by the control unit 210 in thiscase will be explained with reference to FIG. 2.

FIG. 2 is a function block diagram showing functions realized by thecontrol unit 210. FIG. 2 shows a function configuration necessary forrealizing a function of generating a flash image from images picked upby the continuous image-pickup function. In this case, as shown in FIG.2, the control unit 210 functions as a operation mode processing unit211, an image-pickup control unit 212, a picked-up image processing unit213, an object detecting unit 214, an image selecting unit 215, abackground image generating unit 216, a flash image generating unit 217,and the like.

The operation mode processing unit 211 displays a screen necessary forcausing the user to specify various operation modes of the digitalcamera 1 together with the display unit 310, and displays a settingscreen for each operation mode. The operation mode processing unit 211recognizes an operation mode specified by the user together with theoperation unit 330, reads out a program or an arithmetic expressionnecessary for executing such an operation mode from the memory unit 250or the like, and loads the read-out program or expression in the mainmemory device (memory).

In the embodiment, it is assumed that an operation mode relating to aflash image generating function (flash image generating mode) isselected by a user. Each function configuration of the control unit 210to be described below is a function configuration realized as theoperation mode processing unit 211 runs a program loaded in accordancewith selection of the flash image generating mode.

The image-pickup control unit 212 executes an image pickup operation ofthe digital camera 1 by controlling the image pickup unit 100. Accordingto the flash image generating function of the embodiment, because aflash image is generated from picked-up images acquired by thecontinuous image-pickup function of the digital camera 1, theimage-pickup control unit 212 of the embodiment controls the imagepickup unit 100 to perform a continuous image-pickup operation. In thiscase, while the shutter button of the operation unit 330 is beingpressed, the image pickup unit 100 continuously performs image pickupoperations. A picked-up image (continuously-picked-up image) acquiredthrough the continuous image-pickup operation under the control of theimage-pickup control unit 212 is processed by the image processing unit220, and is sequentially stored in the image memory 230. In this case, aframe number is allocated to each continuously-picked-up image stored inthe image memory in the order of image pickup.

The picked-up image processing unit 213 converts and processes acontinuously-picked-up image stored in the image memory 230 in a formataccording to a process relating to generation of a flash image, andexecutes various processes using a picked-up image. In the embodiment,the picked-up image processing unit 213 aligns continuously-picked-upimages, and converts a picked-up image into one-dimensional data inorder to enable high-speed extraction of an object area. Data convertedby the picked-up image processing unit 213 is stored in, for example,the main memory device (memory) of the control unit 210. Note that“one-dimensional data” represents information like a pixel valueconfiguring image data with a value projected on one coordinate axis.For example, a projected value can be calculated by combining pixelvalues, such as an RGB, and a YUB, in a direction perpendicular to theforegoing one coordinate axis. In the embodiment, projection data, whichis acquired by projecting image data in a predetermined direction, formsone-dimensional data. The direction in which image data is projected inthis case is set based on a moving direction of a target object (thiswill be described in detail later).

The object detecting unit 214 detects a part (area) of an object in eachpicked-up image by comparing one-dimensional data among individualpicked-up images. In this case, the object detecting unit 214 detectspositional information (coordinate information) of a detected objectpart. Moreover, the object detecting unit 214 extracts an object partfrom a picked-up image (continuously-picked-up image) stored in theimage memory 230 regarding an image selected by the image selecting unit215.

The image selecting unit 215 selects picked-up images that an objectpart does not overlap one another based on positional information of theobject part detected by the object detecting unit 214, thus selecting anappropriate image (frame) for generation of a flash image among thecontinuously-picked-up images.

The background image generating unit 216 generates a background imagefrom a picked-up image selected by the image selecting unit 215. Thebackground image generating unit 216 acquires pixel values at the samecoordinate for all of the selected picked-up images, sets a centralvalue or the like of the acquired pixel values as a pixel value of thecoordinate, thereby generating a background image in which an object isexcluded.

The flash image generating unit 217 synthesizes an image of the objectpart extracted by the object detecting unit 214 from the selectedpicked-up image with a background image generated by the backgroundimage generating unit 216, thereby generating a flash image that amoving object appears at plural portions in one background image withoutoverlapping one another.

The above-explained functions are ones realized by the control unit 210.Note that in the embodiment, each of the above-explained functions arerealized through a logical process executed by the control unit 210which runs a program, but those functions may be realized by a hardwareresource like an ASIC (Application Specific Integrated Circuit). In thiscase, a part of or all of functions relating to image processing amongthe functions shown in FIG. 2 may be realized by the image processingunit 220.

The above-explained configuration of the digital camera 1 is requisitefor realizing the present invention, and it is presumed that the digitalcamera is provided with a configuration used for a basic function of thedigital camera and various additional functions thereof as needed.

An explanation will be given of an operation of the digital camera 1having the foregoing configuration. An explanation will be given of a“flash image generating process” executed by the digital camera 1 whenthe flash image generating mode is selected with reference to theflowchart of FIG. 3. The flash image generating process starts when theflash image generating mode is selected as the user of the digitalcamera 1 operates the operation unit 330. In this case, the operationmode processing unit 211 loads a program for the flash image generatingmode from the memory unit 250, and each function configuration shown inFIG. 2 executes following processes.

As the process starts, the operation mode processing unit 211 displays asetting screen for a setting necessary for executing the flash imagegenerating mode on the display unit 310 (step S101). FIGS. 4A, 4B showexamples of the setting screen displayed in this step.

First, a setting screen for setting a direction of the camera(horizontal direction and vertical direction) shown in FIG. 4A isdisplayed. This setting screen is for causing the user to select whetherthe digital camera 1 is in a horizontal position or a vertical positionat the time of picking up continuously-picked-up images which will be aflash image. The user of the digital camera 1 operates the operationunit 330, and specifies the direction of the camera at the time ofcontinuous image-pickup. Note that if the digital camera 1 is providedwith a vertical position detecting function realized by, for example, anacceleration sensor, a detection result can be used for such setting, sothat it is not necessary for the user to specify the position throughthe setting screen.

As the direction of the camera is specified, the operation modeprocessing unit 211 displays a setting screen for setting a movingdirection of an object as shown in FIG. 4B on the display unit 310. Thatis, in the flash image generating mode, images of a moving object arepicked up, so that the moving direction thereof is set at this time. Asshown in FIG. 4B, in the setting screen for setting the moving directionof the object, for example, arrows indicating four directions aredisplayed. The moving direction of the object subjected to image pickupis selected as the user operates the cross key or the like of theoperation unit 330.

The operation mode processing unit 211 records the direction of thecamera and the moving direction of the object set in this fashion in themain memory device (memory) or the like of the control unit 210, thuscarrying out the setting for the flash image generating mode (stepS102).

FIG. 5A exemplifies a relationship between the digital camera 1 and theobject when image pickup is carried out in the flash image generatingmode. In this case, it is assumed that images of an object MV movingfrom left to right are picked up. Image pickup is carried out with thedirection of the digital camera 1 and the angle of view thereof beingset at constant. In the example case shown in FIG. 5A, the contents ofsetting in the step S102 are that “the direction of the camera ishorizontal”, and “the moving direction of the object is to right”. Thatis, the setting is made for the case in which the object is moving inthe horizontal direction.

As explained above, images are picked up by continuous image-pickup inthe flash image generating mode, the user of the digital camera 1 makesthe foregoing setting, and then operates the shutter button of theoperation unit 330 to start image pickup. Accordingly, the operationmode processing unit 211 instructs the image-pickup control unit 212 toperform continuous image-pickup when the flash image generating mode isspecified.

As the user (photographer) of the digital camera 1 operates (presses)the shutter button (operation unit 330), the operation unit 330 inputsan input signal according to the user operation into the control unit210. Accordingly, the image-pickup control unit 212 determines that theshutter button is operated (step S103: YES), and controls the imagepickup unit 100 to perform continuous image-pickup (step S104). Thecontinuous image-pickup operation is carried out while the user ispressing the shutter button (step S105: NO).

As the shutter button operation is finished (step S105: YES), theimage-pickup control unit 212 instructs the image pickup unit 100 toterminate the image pickup operation. Accordingly, the continuousimage-pickup operation is terminated, and images (continuously-picked-upimages) acquired by continuous image-pickup are sequentially processedby the image processing unit 220, and stored in the image memory 230(step S106).

FIG. 5B shows examples of continuously-picked-up images acquired when ascene exemplified in FIG. 5A is continuously picked up. As shown in FIG.5B, plural picked-up images express how the object MV moves. Asexplained above, because image pickup is carried out with the directionof the digital camera 1 and the angle of view thereof being set atconstant, there is no substantial change in a part (i.e., a background)other than the object MV in each continuously-picked-up image.

Frame numbers from 1 to p are allocated to individualcontinuously-picked-up images acquired by the continuous image-pickupoperation in the order of image pickup time series. In the example caseof FIG. 5B, images from frame 1 to frame 12 (i.e., p=12) are thusacquired.

As the continuous image-pickup is carried out in this fashion, a processfor generating a flash image using the acquired continuously-picked-upimages is sequentially executed. In this case, as the image-pickupcontrol unit 212 notifies the picked-up image processing unit 213 thatthe continuous image-pickup has been terminated, the picked-up imageprocessing unit 213 starts executing an “alignment process” for aligningthe continuously-picked-up images (step S200). The “alignment process”will be explained with reference to the flowchart of FIG. 6.

As the process starts, the picked-up image processing unit 213 sets aframe (hereinafter, “reference frame N”) which becomes a criteria when adisplacement between the continuous picked-up images is detected amongthe picked-up image frames (1 to p) stored in the image memory 230 (stepS201). In this step, for example, the first frame is set as thereference frame N.

Next, the picked-up image processing unit 213 specifies a frame next tothe reference frame N (i.e., N+1) set in the step S201 to a pointer nspecifying a picked-up image frame (step S202).

The picked-up image processing unit 213 detects a displacement VFbetween the reference frame N and an n-th frame (hereinafter, “frame n”)(step S203). In this case, the picked-up image processing unit 213acquires a vector indicating a position shifting between acharacteristic point set over the reference frame N and thatcharacteristic point over the frame n, thereby acquiring thedisplacement.

The displacement between the frames indicates, for example, a shaking ofa whole picked-up image. As explained above, image pickup to generate aframe image is carried out with the direction of the digital camera 1and the angle of view thereof being set at constant. Accordingly, it ispreferable to carry out image pickup while fixing the digital camera 1by a tripod or the like. However, image pickup may be carried out withthe digital camera 1 being held by hands in some cases. In the case inwhich the digital camera 1 is held by hands, the digital camera 1 may bemoved while continuous image-pickup is carried out. A position shiftingmay be caused in such a case.

In this case, the picked-up image processing unit 213 determines whetheror not the displacement VF detected in the step S203 is smaller than afirst threshold th1 (step S204). Because a flash image is generated bysynthesizing plural object images each indicating a motion of theobject, it is desirable that a background part should be substantiallyconstant across the picked-up plural frames. However, when a positionshifting among the frames is caused because of the movement of thedigital camera 1 as explained above, there is a large difference amongthe frames in the background part. As a result, when a flash image isgenerated in this case, the background part may duplicatingly appear. Inorder to distinguish such a frame, an upper limit value within anacceptable range in which it is possible to presume that the backgroundis constant is set as the first threshold th1.

When the displacement VF between the reference frame N and the frame nis smaller than the first threshold th1 (step S204: YES), there is nolarge position shifting in the frame n relative to the reference frameN. In this case, the picked-up image processing unit 213 specifies theframe n as a frame (effective frame) which can be used for generation ofa flash image (step S207).

In the embodiment, in an “effective frame selecting process” to bedescribed later, an effective frame is selected based on a position ofthe object. Thus, the “effective frame” specified in the “alignmentprocess” is called an “effective frame in a frame level”. Moreover, the“effective frame” specified in the “effective frame selecting process”is called an “effective frame in an object level”.

Conversely, when the displacement VF between the frames is larger thanor equal to the threshold th1 (step S204: NO), the picked-up imageprocessing unit 213 compares the displacement VF with a second thresholdth2 (th1<th2) (step S205). Even if the displacement VF which is largerthan or equal to the first threshold th1 is detected, a positionshifting can be compensated within an acceptable range by performing animage processing which deforms the image of the frame n. Accordingly, anupper limit value within an acceptable range of the displacement VFwhich can be compensated by image deformation is set as the secondthreshold th2.

When the detected displacement VF is smaller than the second thresholdth2 (step S205: YES), the picked-up image processing unit 213 executesan image processing of deforming the image of the frame n to carry outalignment (step S206).

Regarding the frame n which can be aligned by such image deformation,because the difference from the reference frame N is within anacceptable range, the picked-up image processing unit 213 specifies theframe n as an effective frame (in a frame level) which can be used forgeneration of a flash image (step S207).

Note that when the displacement VF between the frames is larger than orequal to the second threshold th2 (step S205: NO), it is difficult toset the difference from the reference frame N to be within an acceptablerange even if image deformation is carried out. In this case, thepicked-up image processing unit 213 specifies the frame n as an invalidframe in order to remove such a frame from a candidate to be synthesizedwhen generating a flash image (step S208).

The picked-up image processing unit 213 which has executed the foregoingprocess creates a “frame information table” shown in FIG. 7 in the mainmemory device (memory) or the like in order to record information on apicked-up frame image. As shown in FIG. 7, the “frame information table”has a record created with a frame number (1 to p) at the time of imagepickup as a key. Each record has information on a corresponding frame.

The picked-up image processing unit 213 which has executed the foregoingprocess records information indicating whether or not a frame is aneffective frame in a frame level and information indicating a comparisonresult of the displacement which is a difference from the referenceframe N with each threshold in the “frame information table” for eachframe. For example, regarding a frame specified in the step S207 as an“effective frame”, “OK” is recorded in a field of an “effective frame(frame level)”, and regarding a frame specified in the step S208 as an“invalid frame”, “NG” is recorded.

After the foregoing processes are performed on the frame n, thepicked-up image processing unit 213 increments the pointer by +1 inorder to specify a next frame (step S209). Moreover, the picked-up imageprocessing unit 213 executes the processes following the step S203 whenthe value of n is smaller than or equal to p which indicates a finalframe of picked-up images (step S210: NO), thus sequentially specifyingwhether or not each frame is an effective frame in a frame level or aninvalid frame.

As the foregoing processes are performed on all frames of the picked-upimages (step S210: YES), the process is terminated and the flow goesback to the “flash image generating process” (FIG. 3).

Note that if the digital camera 1 has a camera-shake compensationfunction, then the digital camera 1 should have the sensor unit (a gyrosensor or the like) 400 which detects a motion of the digital camera 1itself. In this case, if a detection value detected by the sensor unit400 at the time of continuous image-pickup is recorded for each frame,the displacement VF between frames can be acquired based on the recordeddetection value.

As the “alignment process” completes, the picked-up image processingunit 213 executes a “one-dimensional data generating process” forconverting a continuously-picked-up image into one-dimensional data(step S300). The one-dimensional data generating process will beexplained with reference to a flowchart of FIG. 8. In order tofacilitate understanding, it is assumed that the direction of the camerais set as a “horizontal position”.

As the process starts, the picked-up image processing unit 213 sets aninitial value “1” of the frame number to the pointer n (step S301), andselects, as a process target, an image (frame n) having acontinuously-picked-up -image frame number of n stored in the imagememory 230 (step S302).

As a process target image is selected, the picked-up image processingunit 213 initializes a check coordinate in this image (step S303). Inthe embodiment, the picked-up image processing unit 213 takes acoordinate having an x-coordinate and a y-coordinate both of which are“0” as an initial check coordinate. An explanation will be given of acoordinate on an image with reference to FIG. 9A. In the embodiment,because the direction of the camera is set as a “horizontal position”, apicked-up image acquired by the digital camera 1 through such a settingbecomes a horizontally-long image as shown in FIG. 9A. In this case, thehorizontal direction of the image is set as an “X-direction”, and thevertical direction thereof is set as a “Y-direction”. The upper left endof the image is taken as an origin (0, 0). In this case, if the numberof pixels in the X-direction is sizeX, then the maximum value of thex-coordinate becomes sizeX-1. Likewise, if the number of pixels in theY-direction is sizeY, then the maximum value of the y-coordinate becomessizeY-1.

Next, the picked-up image processing unit 213 determines whether themoving direction of the object set in the step S102 in the foregoingflash image generating process (FIG. 3) is the X-direction or theY-direction in the coordinate system of the picked-up image defined asshown in FIG. 9A (step S304). When the direction of the camera is thehorizontal position, if either one of right or left is specified throughthe setting screen shown in FIG. 4B, the moving direction of the objectbecomes the X-direction (the horizontal direction). Conversely, ifeither one of up or down is specified, the moving direction of theobject becomes the Y-direction (the vertical direction).

In the scene exemplified in FIG. 5A, because images of the object MVmoving in the horizontal direction are picked up, it is determined thatthe moving direction of the object is the X-direction (step S304: YES).When the moving direction of the object is the X-direction, thepicked-up image processing unit 213 executes a process of projecting animage in a direction orthogonal to the moving direction of the object,i.e., in the Y-direction.

In this case, the picked-up image processing unit 213 projects an imagein the Y-direction by combining pixel values in the Y-direction at allcoordinates in the X-direction of the process target image (step S305,step S306, and step S307: YES). Because the check coordinate isinitialized in the step S303, first, the picked-up image processing unit213 combines pixel values in y-coordinates corresponding to anx-coordinate 0 in the step S305, and stores the result in the mainmemory device (memory) of the control unit 210. Next, the picked-upimage processing unit 213 increments a value of the x-coordinate by +1,and performs the same calculation for the next x-coordinate in the stepS306. Such a process is repeated by what corresponds to the size of theimage in the X-direction, i.e., the number of pixels in the X-direction.

Conversely, when the moving direction of the object is the Y-direction(vertical direction) (step S304: NO), an image is projected in adirection orthogonal to the moving direction, i.e., the X-directionaccording to the same technique (step S308, step S309, and step S310:YES).

As the projection in the process-target image completes (step S307: NOor step S310: NO), the picked-up image processing unit 213 incrementsthe pointer by +1 (step S311). When the frame number corresponding tothe new pointer n is smaller than or equal to the frame number p of thelast frame (step S312: NO), the picked-up image processing unit 213selects a next continuously-picked-up image as a process target (stepS302).

Conversely, when projection completes for all continuously-picked-upimages (step S312: YES), the process returns to the flow of the flashimage generating process (FIG. 3).

According to such projection through the one-dimensional data generatingprocess, each of the continuously-picked-up images is converted intoone-dimensional data shown in, for example, FIG. 9B, (a) and (b).Generation of such one-dimensional data can be expressed by followingequation 1.

$\begin{matrix}{{{{fn}(x)}\text{:}\mspace{14mu} {projection}\mspace{14mu} {of}\mspace{14mu} {pixel}\mspace{14mu} {value}\mspace{14mu} {{Fn}\left( {x,y} \right)}\mspace{14mu} {in}\mspace{14mu} Y\text{-}{direction}}{{{fn}(x)}:={\sum\limits_{y = 0}^{{sizeY} - 1}{{Fn}\left( {x,y} \right)}}}{{sizeY}\text{:}\mspace{14mu} {pixel}\mspace{14mu} {number}\mspace{14mu} {in}\mspace{14mu} Y\text{-}{direction}}{{{fn}(y)}\text{:}\mspace{14mu} {projection}\mspace{14mu} {of}\mspace{14mu} {pixel}\mspace{14mu} {value}\mspace{14mu} {{Fn}\left( {x,y} \right)}\mspace{14mu} {in}\mspace{14mu} X\text{-}{direction}}{{{fn}(y)}:={\sum\limits_{x = 0}^{{sizeX} - 1}{{Fn}\left( {x,y} \right)}}}{{sizeX}\text{:}\mspace{14mu} {pixel}\mspace{14mu} {number}\mspace{14mu} {in}\mspace{14mu} X\text{-}{direction}}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$

In this fashion, as one-dimensional data (projection data) is createdfor each continuously-picked-up image, the picked-up image processingunit 213 sequentially stores the created one-dimensional data in theimage memory 230 in association with the frame number of an originalpicked-up image. Moreover, the picked-up image processing unit 213notifies the object detecting unit 214 that creation of one dimensionaldata is completed for all continuously-picked-up images. In response tothe notification from the picked-up image processing unit 213, theobject detecting unit 214 executes an “object area detecting process” ofdetecting an object part using converted one-dimensional data (stepS400).

First, a concept of the process of detecting an object part fromone-dimensional data for a continuously-picked-up image will beexplained with reference to FIG. 9B, (a) to (c). FIG. 9B, (a) shows anexample picked-up image (upper part) picked up without the object MV inthe scene shown in FIG. 5A, and an example one-dimensional data (lowerpart) of such a picked-up image. FIG. 9B, (b) shows an example picked-upimage (upper part) picked up with the object MV, and an exampleone-dimensional data (lower part) of such a picked-up image.

When the digital camera 1 of the embodiment generates a flash image, asexplained above, because continuous image-pickup is carried out with thedirection of the digital camera 1 and the angle of view thereof beingset at constant, in the plural acquired continuously-picked-up images,there is no substantial change in the background part except the movingobject MV. Accordingly, when one dimensional data among individualcontinuously-picked-up images are compared one another, there is achange in the range of the object MV. In order to facilitateunderstanding, one-dimensional data of an image having no object MV andthat of an image having the object MV are shown in FIG. 9B, (a) and (b),respectively. When respective one-dimensional data are compared, data ata portion where the object MV appears in FIG. 9B, (b) differs from FIG.9B, (a), while other portions are same.

That is, if one-dimensional data of both images are overlapped with eachother as shown in FIG. 9B, (c), there is a portion which does not matchwith each other. Such a portion where one-dimensional data does notmatch is a range where the object MV appears (a range in the X-directionif the moving direction of the object is X-direction, and a range in theY-direction if it is Y-direction). Accordingly, when a difference inone-dimensional data between adjoining continuously-picked-up images inthe order of time series is acquired, a transition of the object rangecan be expressed.

The object detecting unit 214 executes the process based on such aprinciple, thereby detecting an object part in a continuously-picked-upimage. The object area detecting process executed at this time (stepS400) will be explained with reference to the flowcharts of FIGS. 10 to12.

As the process starts, first, the object detecting unit 214 determineswhether the moving direction of the object is the X-direction or theY-direction based on the setting at the step S102 in the flash imagegenerating process (FIG. 3) (step S401). Hereinafter, an explanationwill be mainly given of a case in which the moving direction is theX-direction (step S401: YES).

In this case, the object detecting unit 214 initializes a coordinate inthe X-direction which is a check target. As explained above, because thecoordinate range of a picked-up image is from (0, 0) to (sizeX-1,sizeY-1), the initial value of an x-coordinate is set to be “0” (stepS402).

Next, the object detecting unit 214 sequentially specifies frames of thecontinuously-picked-up images by specifying the pointer n so as to be 1to p, and for each frame, acquires one-dimensional data corresponding tothe coordinate x generated in the “one-dimensional data generatingprocess” (FIG. 8) from the image memory 230, and stores suchone-dimensional data in, for example, the main memory device (memory) ofthe control unit 210 (step S403 to step S406: NO).

As one-dimensional data for all frames are acquired (step S406: YES),the object detecting unit 214 sorts acquired one-dimensional data (stepS407), and sets a central value thereof to be a pixel value fb(x) thatthe coordinate x indicates a background image (step S408).

The object detecting unit 214 performs such an operation at each pointof the x-coordinate (step S409, and step S410: NO). As background pixelvalues fb(x) for all x-coordinates (0 to sizeX-1) in all frames areacquired (step S410: YES), the process progresses to step S411 (FIG.11).

The object detecting unit 214 sets the pointer n to be a frame initialvalue “1”, and sets the x-coordinate to be a coordinate initial value“0” (step S411, and step S412). Next, the object detecting unit 214calculates a difference fd(x) (=|fb(x)−fn(x)|) between a pixel valuefn(x) at the x-coordinate in an n-th frame and the background pixelvalue fb(x) at that x-coordinate acquired in the step S408 (step S413:YES, and step S414).

The object detecting unit 214 determines whether or not the calculateddifference fd(x) is larger than a threshold DiffTh, thereby determiningwhether or not the coordinate x is a part representing the object (stepS415). That is, because the background pixel value fb(x) is a pixelvalue when the coordinate x represents a background image, if an actualpixel value fn(x) greatly differs from the pixel value of the backgroundimage, the coordinate x in the frame n represents the object MV otherthan the background. Accordingly, the threshold DiffTh which enablessuch determination is set, and if the difference fd(x) is larger thanthe threshold DiffTh, it is possible to determine that the pixel valuefn(x) is a pixel value representing an object part.

When the pixel value fd(x) is smaller than or equal to the thresholdDiffTh (step S415: NO), the coordinate x in the n-th frame is not anobject part. In this case, the object detecting unit 214 increments thex-coordinate by +1 (step S416), and if it is a coordinate within theimage size (step S413: YES), the same determination is performed withrespect to the next coordinate x (step S414, and step S415).

In the initialization in the step S412, because the x-coordinate is setto be “0” which is the left end of the image, when it is determined thatthe coordinate x which is the check coordinate indicates a pixel of theobject part (step S415: YES), the object detecting unit 214 sets suchx-coordinate as a coordinate L(n) which corresponds to the left end ofthe object part in the n-th frame (step S417).

That is, because determination is carried out while incrementing thex-coordinate by +1 from a position where x=0 which is the left end ofthe picked-up image, the process from the step S412 to the step S416 isa searching of the object part from the left end of the image. Becausethe object range for the x-coordinate has a width, it is necessary tospecify the right end of the object range. Accordingly, as the left endof the object part is specified, the object detecting unit 214 performsan operation of searching the right end of the object part.

In this case, the object detecting unit 214 starts searching from theright end side of the picked-up image. Accordingly, the object detectingunit 214 sets a check coordinate x as sizeX-1 which is an x-coordinateindicating the right end of the picked-up image (step S418). Because theright end of the object part is a subject of searching, the checkcoordinate x must be located at right from the coordinate L(n) which isset as the left end of the object part in the step S417. Thus, when thecheck coordinate x is greater than L(n) (step S419: YES), the objectdetecting unit 214 calculates fd(x) for such coordinate x (step S420),and determines whether or not the calculated fd(x) is larger than thethreshold DiffTh (step S421).

When fd(x) at the coordinate x is smaller than or equal to the thresholdDiffTh, and does not indicates the object part (step S421: NO), theobject detecting unit 214 decrements the check coordinate x by −1 toshift the check coordinate to left by what corresponds to 1 coordinate(step S422), and performs the same determination on such coordinate x(step S419, step S420, and step S421). In this fashion, the searching iscarried out from the right end of the picked-up image, and when it isdetermined that a pixel at the check coordinate x indicates the objectpart (step S421: YES), the object detecting unit 214 sets such acoordinate x as a coordinate R(n) which is the right end of the objectpart in the frame n (step S423).

Note that when the object part is not detected while searching isstarted from the left end of the image (step S413: NO), and when thex-coordinate set from the right end is located at left from the objectleft end L(n) (step S419: NO), it is determined that there is no objectpart in the frame n, and for example, the value is set in such a waythat L(n)=R(n), and a check target is shifted to the next frame (stepS424).

Moreover, when the object range is detected for the frame n, theforegoing process is also carried out with the next frame being as anext check target (step S424, and step S425: NO). When the object areais detected for all frames (step S425: YES), the process returns to the“flash image generating process” (FIG. 3).

This is how the object area is detected when the moving direction of theobject is the X-direction. In this case, as shown in FIG. 13A, the rangeof the object MV appearing in each frame in the X-direction issequentially acquired. The coordinates (L(n) and R(n)) of the objectrange acquired in this fashion are stored in the main memory device(memory) or the like in association with the frame number.

Conversely, when the moving direction of the object is the Y-direction,the same process as explained above is executed in the Y-direction. Thatis, through the step S426 to the step S434 in FIG. 10 and the step S435to the step S449 in FIG. 12, an top end coordinate T(n) of the objectimage appearing in each frame and a bottom end coordinate B(n) thereofare acquired (see FIG. 13B).

As the object area is detected in this fashion, the object detectingunit 214 notifies the image selecting unit 215 that the object area isdetected. In this case, the image selecting unit 215 executes the“effective frame selecting process” (step S500 in FIG. 3) for selectinga picked-up image (effective frame) used for generation of a flash imagebased on the detected object area. The effective frame selecting processwill be explained with reference to the flowcharts of FIGS. 14 and 15.

As the process starts, first, the image selecting unit 215 determineswhether the moving direction of the object is the X-direction or theY-direction (step S501). The flowchart of FIG. 14 shows a process in acase in which the moving direction of the object is the X-direction.FIG. 15 shows a process in a case in which the moving direction is theY-direction.

When the moving direction of the object is the X-direction (step S501:YES), the image selecting unit 215 sets the pointer n to be the frameinitial value “1” (step S502). Next, the image selecting unit 215determines whether or not the n-th frame (frame n) is specified as aneffective frame in a frame level based on the process result in theforegoing “alignment process” (step S503). In this case, the imageselecting unit 215 refers to the “frame information table” (FIG. 7)created in the main memory device (memory) of the control unit 210,thereby determining whether or not the frame n is specified as aneffective frame in the frame level.

When the frame n is specified as an “invalid frame” in the frame level(step S503: NO), the background part is largely shifted from thereference frame N, so that if such a frame n is used for generation of aflash image, the background part may be duplicated. Accordingly, theimage selecting unit 215 specifies such a frame n as an invalid framealso in this step (step S504).

In this case, according to the example of the “frame information table”shown in FIG. 7, for example, like the record of the frame number “3”,“NG” is recorded in both fields of “effective frame (frame level)” and“effective frame (object level)”.

Conversely, when the frame n is specified as an “effective frame” in theframe level (step S503: YES), the image selecting unit 215 determineswhether or not the flam n contains an object area based on a detectionresult from the foregoing “object area detecting process” (step S505).When L(n) and R(n) (or T(n) and B(n)) indicating the range of the objectarea are recorded with respect to the frame n and L(n)≠R(n) (orT(n)≠B(n)) is satisfied, the image selecting unit 215 determines thatthe object area is detected from the frame n.

When no object area is detected from the frame n (step S505: NO), theimage selecting unit 215 specifies the frame n as an invalid frame whichis not used for generation of a flash image (step S504).

When the frame n is specified as the “invalid frame” in the step S504,the image selecting unit 215 increments the pointer n by +1 to specifythe next frame (step S510). When the value of n is smaller than or equalto the number of the last frame p (step S511: NO), the flow returns tothe step S503 and the process following the step S503 is repeated.

The image selecting unit 215 sequentially checks presence/absence of theobject area in a frame specified as an effective frame in the framelevel in this fashion. The image selecting unit 215 selects a frame thatis determined as having the object area at first as an effective frame(step S505: YES, step S506: NO, and step S508). The image selecting unit215 allocates an effective frame number which is different from theframe number to the frame (continuously-picked-up image) selected as theeffective frame. In this case, the image selecting unit 215 allocates aneffective frame number “1” to the frame which is selected as a firsteffective frame. Moreover, the image selecting unit 215 sets the framenumber of the frame selected in the step S508 to a pointer m forspecifying an effective frame (step S509).

The image selecting unit 215 records information indicating whether ornot the frame is an effective frame in the frame level and the effectiveframe number allocated to the effective frame (object level) in the“frame information table” (FIG. 7).

First, the image selecting unit 215 selects a first effective frame, andsets the frame number thereof to the pointer m. Next, the imageselecting unit 215 increments the pointer n by +1 to specify the nextframe (step S510), and when the value of n is smaller than or equal tothe number of the final frame p (step S511: NO), executes the processfollowing the step S503. That is, the image selecting unit 215 searchesa next frame to be used for generation of a flash image.

After the first effective frame is selected (step S506: YES), the imageselecting unit 215 compares the position of the object area between acheck target frame (n-th frame) and an effective frame (frame specifiedby pointer m) selected most recently, thereby selecting a frame havingthe object part which does not overlap the previous effective frame asan effective frame.

When the moving direction of the object is the X-direction, there aretwo movements: one from left to right; and another from right to left.When the object moves from left to right, the object area in the frame nmust be located at right from the object area in the frame m.Conversely, when the object moves from right to left, the object area inthe frame n must be located at left from the object area in the frame m.

Accordingly, the image selecting unit 215 determines that no object partof the frame n overlaps the object part of the frame m when the objectleft end L(n) in the frame n is larger than the object right end R(m) inthe frame m or when the object right end R(n) in the frame n is smallerthan the object left end L(m) in the frame m (step S507).

When the object area of the frame n satisfies such a condition (stepS507: YES), the image selecting unit 215 selects the n-th frame as aneffective frame (step S508). Conversely, when the object area of theframe n does not satisfy such a condition (step S507: NO), the imageselecting unit 215 specifies the n-th frame as an invalid frame not usedfor generation of a flash image (step S504).

Regarding following continuously-picked-up images, selection of aneffective frame based on such a determination condition is carried out,and as all continuously-picked-up images are checked and effectiveframes are selected (step S511: YES), as shown in FIG. 16, an effectiveframe to which an effective frame number different from the frame numberis allocated and an invalid frame not used for generation of a flashimage are distinguished.

Note that as shown in FIG. 16, the effective frame number is acontinuous number in the order of time series in selected effectiveframes. In this case, it is assumed that the number of selectedeffective frames is p′ (p′≦p). By allocating identification informationindicating a selected effective frame in this fashion, it is possible todistinguish an effective frame among the continuously-picked-up imagesstored in the image memory 230. However, only effective frames may beleft in the image memory 230 by erasing invalid frames from the imagememory 230.

This is the operation of selecting an effective frame when the objectmoves in the X-direction. An effective frame can also be selectedthrough the same process when the moving direction of the object is theY-direction (step S501: NO in FIG. 14). That is, by executing theprocess from step S512 to step S521 in the flowchart of FIG. 15, frameshaving images of the object moving in the Y-direction do not overlap oneanother can be selected as effective frames. In this case, regarding aselection condition after a first effective frame is selected, a case inwhich the object moves from up to down and a case in which the objectmoves from down to up are taken into account.

That is, when the object moves from up to down, the object area in theframe n must be located under the object area in the effective frame mselected most recently. Conversely, when the object moves from down toup, the object area in the frame n must be located above the object areain the effective frame m selected most recently.

Accordingly, the image selecting unit 215 determines that no object partof the frame n overlaps the object part of the effective frame mselected most recently when the object top end T(n) in the frame n islarger than the object bottom end B(m) or when the object bottom endB(n) in the frame n is smaller than the object top end T(m) in the framem selected most recently (step S517 in FIG. 15).

As explained above, the object area is detected and frames which do nothave an overlapped object area are selected from one-dimensional dataacquired by projecting a picked-up image in one direction. That is, asan image (effective frame) used for generation of a flash image isselected based on one-dimensional data having a little data volume, theworkload can be dramatically reduced in comparison with a case of usingall data of picked-up images.

As an image (effective frame) used for generation of a flash image isselected in this fashion, in the “flash image generating process” (FIG.3), a “background image generating process” (step S600) for generating abackground image for the flash image and an “object image extractingprocess” (step S700) for extracting an image of the object part from aneffective frame are sequentially executed using the image data of theselected frame.

The “background image generating process” will be explained withreference to the flowchart of FIG. 17. The background image generatingprocess is started as the image selecting unit 215 notifies thebackground image generating unit 216 that the effective frame selectingprocess completes.

As the process starts, the background image generating unit 216initializes a coordinate (x, y) to be checked over an effective frame.Both x and y are set to be the initial value “0”, so that the coordinateorigin (0, 0) of the effective frame is set to be a check coordinate(step S601).

Next, the background image generating unit 216 causes a pointer n′ whichspecifies an effective frame to sequentially specify 1 to p′, thussequentially acquiring pixel values of the set coordinate (x, y) fromall effective frames (1 to p′) stored in the image memory 230 (step S602to step S604, and step S605: NO).

As the pixel values of the coordinate (x, y) are acquired from alleffective frames (step S605: YES), the background image generating unit216 sorts the acquired pixel values (step S606), and sets the centralvalue thereof as a pixel value (background pixel value fb′(x, y)) thatthe coordinate (x, y) indicates a background part (step S607).

As a background pixel value fb′(x, y) for the coordinate (x, y) isacquired, the background image generating unit 216 sequentially setscoordinates in the image range of an effective frame as checkcoordinates, and acquires a background pixel value fb′(x, y) for thecoordinate (x, y) (step S608, step S609: NO, and the step S602 to thestep S607). For example, the x-coordinate is successively incremented by+1 for the same y-coordinate, the y-coordinate is incremented by +1 andthe x-coordinate is initialized to 0 when the x-coordinate reaches theright end of the image, thereby sequentially acquiring pixel values from(0, 0) to (sizeX-1, sizeY-1), and acquiring background pixel valuesfb′(x, y) for individual coordinates.

As the background pixel values fb′(x, y) are acquired from allcoordinates (step S609: YES), the background image generating unit 216sets the acquired background pixel value fb′(x, y) as a pixel value ateach coordinate, and generates a background image representing only abackground part in the effective frame (step S610). The background imagegenerating unit 216 stores the generated background image in the imagememory 230, and notifies the object detecting unit 214 that thebackground image is created to terminate the process.

In this case, the process returns to the flash image generating process(FIG. 3), and the “object image extracting process” (step S700) is thenexecuted. The object image extracting process is executed by the objectdetecting unit 214 as the background image is generated. The objectimage extracting process will be explained with reference to theflowchart of FIG. 18.

As the process starts, first, the object detecting unit 214 initializesa coordinate (x, y) (step S701) and initializes a target frame (stepS702). Next, the object detecting unit 214 sequentially acquires a pixelvalue fn′(x, y) at a coordinate (x, y) from each of the effective frames1 to p′, and calculates a difference fd′(x, y) between such a pixelvalue fn′(x, y) and the background pixel value fb′(x, y) at thatcoordinate (x, y) (step S703 to step S705, and step S706: NO).

As differences fd′(x, y) at the coordinate (x, y) are acquired from alleffective frames (step S706: YES), the object detecting unit 214calculates a standard deviation fs(x, y) of the calculated differencesfd′(x, y) (step S707). In the embodiment, for example, the objectdetecting unit 214 operates equation 2 to acquire the standard deviationfs(x, y).

$\begin{matrix}{{{fs}\left( {x,y} \right)} = \sqrt{\frac{\sum\limits_{n^{\prime} = 1}^{p^{\prime}}\left( {{fd}^{\prime}\left( {n^{\prime},x,y} \right)}^{2} \right)}{p^{\prime} - 1}}} & \left\lbrack {{Equation}\mspace{14mu} 2} \right\rbrack\end{matrix}$

where fd′(n′, x, y) is a difference fd′(x, y) at the coordinate (x, y)in a frame n′.

As the standard deviation fs(x, y) at the coordinate (x, y) is acquired,the object detecting unit 214 shifts the check coordinate within theimage range of the effective frame, and sequentially acquires a standarddeviation fs (x, y) at each coordinate (step S708, step S709: NO, thestep S702 to the step S707).

As the standard deviation fs(x, y) at each coordinate is acquired (stepS709: YES), the object detecting unit 214 sets a variable threshold moveused for determination of the object part in each effective frame basedon the acquired standard deviations fs(x, y) (step S710). In theembodiment, for example, the object detecting unit 214 operates equation3 to set the variable threshold move.

$\begin{matrix}{{{over}\lbrack m\rbrack} = {{{fd}^{\prime}\left( {n^{\prime},x,y} \right)}\left\{ {{\begin{matrix}{{{fs}\left( {x,y} \right)} < {{fd}^{\prime}\left( {n^{\prime},x,y} \right)}} \\{1 \leq m \leq {maxm}}\end{matrix}{move}} = {{\sqrt{\frac{\sum\limits_{m = 1}^{maxm}\left( {{over}\lbrack m\rbrack}^{2} \right)}{{maxm} - 1}}{maxm}\text{:}\mspace{14mu} {number}\mspace{14mu} {of}\mspace{14mu} {pixels}\mspace{14mu} {satisfying}\mspace{14mu} {{fs}\left( {x,y} \right)}} < {{fd}^{\prime}\left( {n^{\prime},x,y} \right)}}} \right.}} & \left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack\end{matrix}$

As the variable threshold move is set, the object detecting unit 214acquires a pixel value which satisfies a difference fd′(x, y)≧variablethreshold value move from each of the effective frames (1 to p′) storedin the image memory 230 (step S711, and step S712).

The object detecting unit 214 sets a pixel value satisfying such acondition as a pixel value indicating an object part in the frame n′(step S713), and performs an “image smoothing process” on the objectpart indicated by that pixel value in order to compensate a deficit ofan image (step S900). The “first image smoothing process” according tothe embodiment will be explained with reference to the flowchart of FIG.19.

As the process starts, the object detecting unit 214 refers toinformation on a displacement VF between the frame n′ and the referenceframe N from the “frame information table” (FIG. 7) created in the mainmemory device (memory) of the control unit 210 (step S911), anddetermines whether or not the displacement VF is less than the firstthreshold th1 (step S912).

As explained above, because the first threshold th1 is the upper limitvalue of the acceptable range of the displacement VF that the backgroundcan be presumed as constant between frames, when the displacement VFfrom the reference frame N is smaller than the first threshold th1, adifference between the frames is small, so that a difference in thebackground part is also small.

Accordingly, in such a case (step S912: YES), the object detecting unit214 performs morphological operation for repairing a deficit of theextracted object area and smoothing it at a normal workload. In theembodiment, for example, the object detecting unit 214 performsmorphological dilation (step S913), performs morphological erosion twice(step S914), and then performs morphological dilation again (step S915).

Conversely, when the displacement VF is larger than or equal to thefirst threshold th1 (step S912: NO), alignment by image deformation hasbeen performed on the frame n′ in the foregoing “alignment process”(FIG. 6). In this case, because image deformation has been carried out,there may be a difference between an original image and a deformed imageat the background part, and such a different part may be falselyrecognized as the object MV.

In particular, when an image of a scene having a deep depth asexemplified in FIG. 20, (a) is picked up, because of the effect ofso-called occlusion, a difference in the background when the entireframe image moves tends to be large.

For example, when a frame N+1 next to the frame image shown in FIG. 20,(a) is an image as shown in FIG. 20, (b), because of a slight movementof the digital camera due to image pickup with the digital camera 1being held by hands, a fluctuation occurs in the background part whichmust be originally constant, and if the displacement VF is larger thanor equal to the first threshold th1, alignment by image deformation isperformed on the frame N+1 in the foregoing “alignment process” (FIG.6).

In this case, as shown in FIG. 20, (c), the frame N has an originalbackground image, but as shown in FIG. 20, (d), the frame N+1 has adeformed background image. Because alignment by image deformation hasbeen carried out, even if the position of the background image of theframe N and that of the background image of the frame N+1 is same, asshown in FIG. 20, (e), a difference in the shape or the like of thebackground image is caused.

In the case of the scene having a deep depth as shown in FIG. 20, such adifference in the shape of the background part becomes a largedifference because of the effect of occlusion. If a background image andobject areas are extracted from such images to synthesize a flash image,as shown in FIG. 20, (f), a partial difference may appear in thebackground part. Because the background part must be originallyconstant, such a part is an unnecessary background image.

When such an unnecessary part is present in the vicinity of the objectMV on a frame image, as shown in FIG. 20, (g), such an unnecessary partmay be falsely recognized as the object MV and extracted. Smoothing bythe foregoing morphological operation is for repairing a deficit of theobject image and compensating it, so if the amount of operation isincreased at that time, the falsely-recognized background part shown inFIG. 20, (g) can be erased.

However, if the amount of operation is equally increased, the workloadincreases, resulting in a longer process time for generating a flashimage. Moreover, when an area which must be originally recognized as theobject MV (dynamic body) is relatively small, such an area may also beerased. Accordingly, if the weight of the smoothing process is changedfor a frame which is likely to be affected by occlusion and for a framenot likely to be affected, an area having a background part falselyrecognized as the object (dynamic body) can be effectively eliminated.

Therefore, in the process, it is determined whether or not a targetframe is likely to be affected by occlusion based on the displacement VFbetween frames, and the amount of process of morphological operation ischanged based on the determination result.

Thus, when the displacement VF between frames is larger than or equal tothe first threshold th1 (step S912: NO), the object detecting unit 214increases the amount of process of morphological operation, therebyeliminating an unnecessary part present in the vicinity of the objectMV. In this case, the object detecting unit 214 performs morphologicaloperation having an increased amount of operation more than a generalprocess amount on the target flam n′. In the embodiment, for example,the object detecting unit 214 performs morphological dilation (stepS916), performs morphological erosion four times (step S917), and thenperforms morphological dilation again (step S918).

In the case of the embodiment, morphological erosion having the amountof process twice as much as a normal morphological operation isperformed on a frame which is determined as a frame likely to beaffected by occlusion based on the displacement VF between frames. Byincreasing the amount of process (number of times) of the morphologicalerosion, a tendency of eliminating a minute difference can be enhanced,and even if a background part is falsely recognized as the object(dynamic body) because of occlusion, such an area can be eliminated.

As morphological operation is performed in this fashion based on thedisplacement VF between frames, the object detecting unit 214 extracts asmoothed object area as a final object image (step S919). In this case,for example, the object detecting unit 214 generates data in which acoordinate that satisfies a difference fd′(x, y)≧variable threshold movebecomes “1”, and in which a coordinate that satisfies a differencefd′(x, y)<variable threshold move becomes “0”. With respect to generateddata for 0 and 1, for example, labeling of allocating a unique number toeach of continuous areas to which a value “1” is allocated is performed.A pixel value of an area which becomes the maximum among the areashaving undergone labeling is set to be a pixel value indicating anobject part in the frame n′.

In the foregoing “first smoothing process”, the amount of process ofmorphological operation is changed based on a comparison with the firstthreshold th1 used in the foregoing “alignment process” (FIG. 6).Because the criterion for setting the amount of process is thedisplacement between frames, the threshold for comparison is not limitedto the first threshold th1 only. That is, a larger number of thresholdscan be set, and the amount of process can be changed more finely bycomparing the intra-frame displacement VF with those thresholds. In anycases, the larger the displacement between frames is, the more anunnecessary part generated due to the effect of occlusion can beeffectively erased by increasing the amount of process of morphologicaloperation.

When an object image that an unnecessary part is eliminated by the“first smoothing process” is extracted from the frame n′, the processreturns to the “object image extracting process” (FIG. 18). The pointern′ is incremented by +1 to specify the next effective frame (step S714),and if the n′-th effective frame is a frame present in the image memory230 (step S715: NO), the process following the step S712 is executed.The object image is extracted from each effective frame in this fashion.

Through the foregoing process, as object images are extracted from alleffective frames stored in the image memory 230, the object detectingunit 214 sends a notification to that effect to the flash imagegenerating unit 217.

The flash image generating unit 217 synthesizes the background imagegenerated through the “background image generating process” (step S600)in the “flash image generating process” (FIG. 3) with the object imagein each effective frame extracted through the “object image extractingprocess” (step S700), thereby executing an “image synthesizing process”(step S800) for generating a flash image as shown in FIG. 22.

The “image synthesizing process” will be explained with reference to theflowchart of FIG. 22. The image synthesizing process is started as theobject detecting unit 214 notifies the flash image generating unit 217that the object image extracting process completes.

As the process starts, the flash image generating unit 217 initializes atarget frame (step S801), and sequentially superimposes object imagesextracted from individual effective frames 1 to p′ through the “objectimage extracting process” (step S700) to synthesize object areas of aflash image (step S802 to step S804).

As all object image areas of the flash image are synthesized (step S802:YES), the flash image generating unit 217 superimposes the backgroundimage generated through the “background image generating process” (stepS600) to synthesize a background area of the flash image (step S805).Note that the background area of the flash image may be synthesized bysuperimposing any one image of the individual effective frames 1 to p′.

As the flash image is generated through the foregoing process, the flashimage generating unit 217 stores the generated flash image in the imagememory 230, and notifies the object detecting unit 214 that the flashimage is generated to terminate the process.

In this case, the flow returns to the flash image generating process(FIG. 3), the generated flash image is displayed on the display unit 310through the image output unit 240, and in accordance with an operationinput as the user of the digital camera 1 operates the operation unit330, the flash image is stored in the memory unit 250 or the externalmemory unit 260 (step S107). The process is then terminated.

Note that when the direction of the camera is the vertical direction, ifthe foregoing process is executed in a coordinate system in which thepicked-up image is rotated by −90 degree or +90 degree, a flash imagecan also be generated.

(Second Embodiment) An explanation will be given of another exampleoperation of the “image smoothing process” (FIG. 18, step S900) executedfor eliminating an unnecessary part from an object image. In the “firstimage smoothing process” of the first embodiment, by comparing adisplacement VF between frames with a threshold, a two-value maskprocess which changes the number of execution times of morphologicalerosion between twice and four times is executed. The number ofexecution times of morphological erosion can be more finely controlledin accordance with a displacement VF. A “second image smoothing process”executed in this case will be explained with reference to the flowchartof FIG. 23.

As the process starts, like the “first image smoothing process” of thefirst embodiment, the object detecting unit 214 accesses the “frameinformation table” (FIG. 7) created in the main memory device (memory)of the control unit 210, and refers to information on a displacement VFbetween a frame n′ which is an extraction target frame for an objectimage and the reference frame N (step S921).

As the displacement VF is referred, the object detecting unit 214performs calculation for setting the number of execution times ofmorphological erosion based on the displacement VF (step S922). Thedisplacement VF is multiplied by a coefficient k (e.g., a positiveinteger greater than or equal to 1) to calculate the number of executiontimes L of morphological erosion (L=VF×k).

As the number of execution times L of morphological erosion iscalculated, the object detecting unit 214 performs morphologicaldilation one time (step S923).

Next, as morphological dilation is performed one time, the objectdetecting unit 214 performs morphological erosion L times which is thenumber of execution times calculated at the step S922 (step S924).

Thereafter, as morphological erosion is performed L times, the objectdetecting unit 214 performs morphological dilation one time (step S925),and completes morphological operation for smoothing of an image.

As the morphological operation completes, the object detecting unit 214extracts the smoothed object area as a final object image (step S926),and terminates the process (return to the flow of the “object imageextracting process”).

In the “second image smoothing process”, by multiplying the displacementVF by the coefficient, the number of execution times of morphologicalerosion is calculated. Accordingly, the amount of process ofmorphological operation (morphological erosion) can be increased inproportion to the displacement between frames, resulting in furthereffective elimination of an unnecessary part which is likely to begenerated because of occlusion.

In the “first image smoothing process” (first embodiment) and the“second image smoothing process” (second embodiment), as an example ofmorphological operation, the explanation has been given of a case inwhich morphological dilation, morphological erosion, and morphologicaldilation are performed in this order. However, an operation in whichmorphological erosion, morphological dilation, and morphological erosionare performed in this order may be added.

That is, in either case in which, as an operator defining the content ofmorphological operation, “Opening” (execution order: dilation; erosion;and dilation) is set or in which “Closing” (execution order: erosion;dilation; and erosion) is set, by changing the amount of process (numberof execution times) in accordance with the displacement VF, anunnecessary part generated because of occlusion can be effectivelyeliminated.

In this case, it is optional to set the amount of process (number ofexecution times) for morphological erosion to which time ofmorphological erosion. For example, the number of execution times of allmorphological erosion can be set to be larger than that of morphologicaldilation, and the number of execution times of morphological erosiononly when “Opening” is set can be set to be larger than that ofmorphological dilation.

In the embodiment, the amount of process (number of execution times) ofmorphological erosion is increased in order to enhance a tendency ofeliminating a minute difference. However, an unnecessary part can alsobe effectively erased by increasing the amount of process (number ofexecution times) of morphological dilation. That is, a target having anamount of process changed in accordance with the displacement betweenframes can be both of morphological erosion and morphological dilation.

(Third Embodiment) In the “image smoothing process” in each of theforegoing embodiments, the explanation has been given of the examplecase in which the number of execution times of morphological operationis changed in accordance with a displacement VF. The amount of processto be changed is not limited to the number of execution times. Anunnecessary part generated because of occlusion can also be effectivelyeliminated by changing the effect (i.e., the amount of erosion ordilation) of morphological operation in accordance with the displacementbetween frames.

That is, in morphological operation, erosion an dilation are carried outby chopping a target image with a “structuring element” having a simpleshape or by adding such a structuring element to the target image, so ifthe size of the structuring element is changed, the amount of erosion ordilation can be changed.

In the embodiment, an explanation will be given of an example case inwhich an unnecessary part is eliminated through such a method. A “thirdimage smoothing process” executed in this case will be explained withreference to the flowchart of FIG. 24.

As the process starts, like the “first image smoothing process” in thefirst embodiment, the object detecting unit 214 accesses the “frameinformation table” (FIG. 7) created in the main memory device (memory)of the control unit 210, and refers to information on a displacement VFbetween a frame n′ which is an extraction target frame for an objectimage and the reference frame N (step S931).

In the embodiment, like each of the foregoing embodiments, the amount ofprocess of morphological operation is changed in accordance with thedisplacement VF between frames to effectively eliminate an unnecessarypart. In this case, according to the embodiment, by changing the effectof morphological operation, i.e., the amount of erosion or dilation, theamount of process for smoothing of an image is changed.

In the third image smoothing process, like each of the foregoingembodiments, operation is performed in the order of morphologicaldilation, morphological erosion, and morphological dilation. In order toenhance a tendency of eliminating a minute difference, a target havingthe amount of process to be changed is set to be morphological erosion.That is, the amount of image shrinkage by one morphological erosion isincreased.

In order to execute such a process, in the embodiment, the size of astructuring element used for morphological dilation (hereinafter,“structuring element SEd”) and that of a structuring element used formorphological erosion (hereinafter, “structuring element SEe”) differfrom each other.

Because morphological dilation is set to have a normal process amount,the size of the structuring element SEd is set to be a default sizethereof (hereinafter, “size DS”). In this case, the object detectingunit 214 sets the size DS as the size of the structuring element SEd toan operator defining the operation content of morphological dilation(step S932).

Conversely, because the amount of process of morphological erosion ischanged in accordance with the displacement VF, the object detectingunit 214 calculates the size of the structuring element SEe based on thedisplacement VF referred at the step S931 (step S933). In theembodiment, by multiplying the default size DS of the structuringelement by the displacement VF, an enlarged size (hereinafter, “enlargedsize LS” or “size LS”) of the structuring element is calculated(LS=VF×DS).

The object detecting unit 214 sets the size LS as the size of thestructuring element SEe to the operator defining the content ofmorphological erosion (step S934). For example, the size LS isrepresented by a positive number, and the larger the set value is, thelarger the size of the structuring element becomes. That is, the largerthe displacement VF between frames is, the larger the size of thestructuring element used for a predetermined morphological operationbecomes.

As the size of the structuring element is set to the operator for eachoperation, the object detecting unit 214 performs operation in the orderof “morphological dilation, morphological erosion, and morphologicaldilation” based on such an operator (step S935, step S936, and stepS937).

In this case, because the size of the structuring element SEe used formorphological erosion is the size LS which is acquired by enlarging thesize DS set for the structuring element SEd used for morphologicaldilation in accordance with the displacement VF, image erosion iscarried out with a larger level than the dilation amount at the time ofcarrying out morphological dilation.

As such a morphological operation completes, the object detecting unit214 extracts a smoothed object area as a final object image (step S938),and terminates the process (return to the flow of the “object imageextracting process” (FIG. 18)).

In the “third image smoothing process”, the enlarged size LS iscalculated by multiplying the default size (size DS) of the structuringelement used for a morphological operation by the displacement VF, it ispossible to increase the amount of process (the amount of dilation orerosion) of the morphological operation in proportion to thedisplacement between frames. Accordingly, it is possible to furthereffectively eliminate an unnecessary part generated because ofocclusion.

In the embodiment, as an example of the morphological operation, anoperation is carried out in the order of “morphological dilation,morphological erosion, and morphological dilation”. However, anoperation carried out in the order of “morphological erosion,morphological dilation, and morphological erosion” may be added.

That is, even if either one of “Opening” (execution order: dilation;erosion; and dilation) or “Closing” (execution order: erosion; dilation;and erosion) is set to the operator, it is possible to effectivelyeliminate an unnecessary part generated because of occlusion by changingthe amount of process (the size of structuring element) in accordancewith the displacement VF.

In this case, it is optional to which time morphological erosion theamount of process (the changed size of structuring element) formorphological erosion is applied. For example, the size of structuringelement used for all morphological erosion may be set large. Moreover,the structuring element having a larger size than morphological dilationmay be used for morphological erosion carried out only when “Opening” isset.

In the embodiment, the amount of process of morphological erosion isincreased (the size of structuring element is enlarged) in order toenhance a tendency of eliminating a minute difference. However, anunnecessary part can be also effectively eliminated by increasing theamount of process of morphological dilation (enlarging the size ofstructuring element). That is, the target having the amount of processto be changed based on the displacement between frames can be bothmorphological erosion and morphological dilation.

As explained above, as the present invention is applied in accordancewith the foregoing embodiments, an effect that a background phase variesbecause of occlusion can be suppressed in the smoothing process at thetime of generating a flash image, thereby suppressing any duplication ofthe background part.

That is, when an image with a background having a deep depth isgenerated by continuous image-pickup, if a displacement between framescaused because of a slight movement of the camera at the time of imagepickup is large, a difference in the background part becomes largebecause of occlusion, and such a background part may be falselyrecognized as a moving object. Accordingly, when the displacementbetween frames is large, for example, the area which is falselyrecognized as the moving object can be eliminated by increasing, forexample, the amount of process of the smoothing process through amorphological operation.

Conversely, when the displacement between frames is small, acomprehensive workload is reduced by not increasing the amount ofprocess of the smoothing process, and even if the area of a movingobject is small, a flash image can be generated without erasing such anarea.

That is, the larger the displacement is, the more the amount of processfor smoothing is increased, resulting in an effective generation of aflash image. As changeover of the amount of process for such a smoothingprocess is carried out based on a comparison of the detecteddisplacement with a threshold, the process becomes efficient.

Alternatively, as the number of execution times of the process forsmoothing can be changed by multiplying the displacement with acoefficient, or as the amount of process for smoothing can be changed bymultiplying the default size of a structuring element with adisplacement, it is possible to generate a good flash image moreeffectively.

Alignment is carried out in such a way that the background part inpicked-up images becomes substantially constant, but when thedisplacement between frames is too large to carry out alignment, such aframe is excluded from a synthesis target for a flash image, so that itis possible to suppress any generation of a flash image having anoverlapped background because of the effect of occlusion.

Moreover, if an image pickup device having a camera-shake compensationfunction is used, a displacement between frames can be acquired from adetection value of a sensor for camera-shake compensation, so that aprocess of calculating the displacement between frames can be reduced,thereby generating a flash image faster.

The foregoing embodiments are merely examples, and the present inventionis not limited to the foregoing embodiments. For example, in theforegoing embodiments, a picked-up image is converted intoone-dimensional data to execute the process in order to detect an objector a background in a flash image. However, how to detect an object or abackground is optional, and is not limited to the above-explainedexample.

Moreover, in the foregoing embodiments, as a criterion for selecting aneffective frame, a condition that an object does not overlap is set.However, a condition to be a criterion for selecting an effective frameis optional, and is not limited to the above-explained example.

Furthermore, in the foregoing embodiments, a displacement between areference frame and a following each frame is detected in executing theimage smoothing process, it is optional that which frame is selected asa frame for acquiring the displacement. For example, a displacementbetween adjoining frames can be acquired.

How to acquire the displacement is also optional, and any schemes whichallow acquisition of a value that allows a determination of whether ornot a frame is likely to be affected by occlusion can be employed.

In the third embodiment, the amount of process of the process forsmoothing is changed by changing the size of a structuring element,factors other than the size can be changed if it results in generation adifference in the effect of the smoothing. For example, if a differencein the shapes of the structuring elements affects the effect of thesmoothing, the shape of a structuring element to be applied can bechanged based on a displacement between frames.

When the present invention is embodied with an image pickup device likethe digital camera 1 exemplified in the foregoing embodiments, an imagepickup device having the configuration of the present invention and thefunctions thereof beforehand can be produced, but an existing imagepickup device can be used as the image pickup device of the presentinvention by applying a program which realizes the same function as eachfunction of the control unit 210 to such an image pickup device.

Although the explanation has been given of the digital still camera asan example of the image pickup device in the foregoing embodiments, theform of the image pickup device is optional. For example, the presentinvention can be applied to a single-unit digital still camera, and canbe applied to various electronic devices (e.g., a mobile phone) havingthe same image-pickup function.

In the foregoing embodiment, the explanation has been given of anexample case in which a flash image is generated from images acquired bycontinuous-image-pickup by a digital still camera. However, a flashimage can be generated from motion picture data because it is ok if aframe image having an object continuously changed is acquired.Therefore, a flash image can be generated at fast if the presentinvention is applied to various image pickup devices having a motionpicture acquiring function like a video camera.

The present invention is not limited to image pickup devices if aplurality of continuous images in which images of a moving object arepicked up can be acquired, and it is possible to generate a flash imagethat the effect of occlusion is reduced by executing the foregoingprocess using various devices (e.g., a personal computer) which canexecute an image processing.

In such a case, by applying the program which realizes the function ofthe present invention, an existing device can function as the imagepickup device of the present invention.

How to apply such a program is optional, for example, such a program canbe applied through a memory medium, such as a CD-ROM or a memory card,storing the program, and the program can be also applied through acommunication medium like the Internet.

If a device which can execute an image processing and to which theforegoing program is applied is used, generation of a flash image at afast speed can be realized. That is, the present invention is notlimited to the image pickup devices, and for example, if the foregoingprogram is applied to a personal computer or the like, a flash image canbe generated at a fast speed from images picked up beforehand.

Various embodiments and changes may be made thereunto without departingfrom the broad spirit and scope of the invention. The above-describedembodiments are intended to illustrate the present invention, not tolimit the scope of the present invention. The scope of the presentinvention is shown by the attached claims rather than the embodiment.Various modifications made within the meaning of an equivalent of theclaims of the invention and within the claims are to be regarded to bein the scope of the present invention.

What is claimed is:
 1. An image processing device comprising: an extracting unit which extracts an image representing a moving object part from picked-up images; a detecting unit which detects a displacement of a background part among the picked-up images; and a smoothing unit which performs a smoothing process on the image representing the moving object part at a smoothing strength in accordance with the displacement of the background part.
 2. The image processing device according to claim 1, wherein the larger the displacement detected by the detecting unit is, the more the smoothing unit increases the smoothing strength of the smoothing process.
 3. The image processing device according to claim 1, wherein the smoothing unit changes a number of execution times of the smoothing process based on a comparison of the displacement detected by the detecting unit with a threshold.
 4. The image processing device according to claim 1, wherein the smoothing unit changes a number of execution times of the smoothing process based on a value acquired by multiplying the displacement detected by the detecting unit by a coefficient.
 5. The image processing device according to claim 1, wherein the smoothing unit performs the smoothing process through a morphological operation, and the smoothing unit changes the smoothing strength of the smoothing process by changing a structuring element used for the morphological operation based on the displacement detected by the detecting unit.
 6. The image processing device according to claim 5, wherein the smoothing unit changes a size of the structuring element based on the displacement.
 7. The image processing device according to claim 1, further comprising a sensor which detects a movement of the image processing device, wherein the detecting unit detects a displacement between frames based on a detection result acquired by the sensor.
 8. The image processing device according to claim 1, further comprising: a generating unit which generates a background image from the picked-up images; and a synthesizing unit which synthesizes the background image with plural smoothened images each representing the moving object part to generate a flash image.
 9. The image processing device according to claim 8, wherein the detecting unit acquires the displacement when the picked-up images are aligned with one another, and the image processing device further comprises an excluding unit which excludes, when the displacement detected by the detecting unit is larger than a predetermined threshold, the picked-up image having the detected displacement from being a candidate of synthesis by the synthesizing unit.
 10. A method of controlling an image processing device, the method comprising: extracting an image representing a moving object part from picked-up images; detecting a displacement of a background part among the picked-up images; and performing a smoothing process on the image representing the moving object part at a smoothing strength in accordance with the displacement of the background part.
 11. A non-transitory computer-readable recording medium having a program stored thereon which is executable by a computer that controls an image processing device, the program controlling the computer to perform functions comprising: extracting an image representing a moving object part from picked-up images; detecting a displacement of a background part among the picked-up images; and performing a smoothing process on the image representing the moving object part at a smoothing strength in accordance with the displacement of the background part. 